Prior-exposure-test Experiment
Sebastian Schuster
10/24/2022
Catch trial performance
## [1] "Excluded 1 participants based on catch-trial performance."## [1] "Excluded participants:"## [1] 1529
Aggregated results
## `summarise()` has grouped output by 'workerid', 'percentage_blue', 'modal'. You
## can override using the `.groups` argument.
## `summarise()` has grouped output by 'percentage_blue', 'modal'. You can
## override using the `.groups` argument.
## `summarise()` has grouped output by 'workerid', 'percentage_blue', 'modal'. You
## can override using the `.groups` argument.
Comparison across conditions
## `summarise()` has grouped output by 'workerid', 'percentage_blue', 'modal'. You
## can override using the `.groups` argument.
## `summarise()` has grouped output by 'percentage_blue', 'modal'. You can
## override using the `.groups` argument.
Individual responses
AUC computation
We use the AUC function with the splines
method to directly compute the AUC.
t-test and regression model with control variables:
##
## Two Sample t-test
##
## data: aucs.cautious$auc_diff and aucs.confident$auc_diff
## t = 3.5648, df = 126, p-value = 0.0005153
## alternative hypothesis: true difference in means is not equal to 0
## 95 percent confidence interval:
## 8.326556 29.108058
## sample estimates:
## mean of x mean of y
## 20.693458 1.976151## Linear mixed model fit by REML. t-tests use Satterthwaite's method [
## lmerModLmerTest]
## Formula:
## auc_diff ~ cond + test_order + first_speaker_type + confident_speaker +
## first_speaker_type * cond + (1 | workerid)
## Data: auc_d
##
## REML criterion at convergence: 1180
##
## Scaled residuals:
## Min 1Q Median 3Q Max
## -2.32868 -0.50839 0.01431 0.64800 1.76492
##
## Random effects:
## Groups Name Variance Std.Dev.
## workerid (Intercept) 335.3 18.31
## Residual 474.6 21.79
## Number of obs: 128, groups: workerid, 64
##
## Fixed effects:
## Estimate Std. Error df t value Pr(>|t|)
## (Intercept) 12.098 3.004 60.000 4.028 0.000161 ***
## cond1 9.489 1.929 62.000 4.918 6.75e-06 ***
## test_order1 1.833 3.010 60.000 0.609 0.544860
## first_speaker_type1 -7.429 2.998 60.000 -2.478 0.016055 *
## confident_speaker1 -5.887 3.007 60.000 -1.958 0.054871 .
## cond1:first_speaker_type1 -2.090 1.929 62.000 -1.083 0.282873
## ---
## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
##
## Correlation of Fixed Effects:
## (Intr) cond1 tst_r1 frs__1 cnfd_1
## cond1 0.000
## test_order1 0.060 0.000
## frst_spkr_1 -0.062 0.000 -0.007
## cnfdnt_spk1 -0.024 0.000 0.092 -0.030
## cnd1:frs__1 0.000 -0.062 0.000 0.000 0.000Clustering analyses
library(mclust)## Package 'mclust' version 5.4.10
## Type 'citation("mclust")' for citing this R package in publications.##
## Attaching package: 'mclust'## The following object is masked from 'package:DescTools':
##
## BrierScore## The following object is masked from 'package:bootstrap':
##
## diabetesaucs_diff = merge(aucs.cautious, aucs.confident, by=c("workerid"))
aucs_diff$diff_of_diffs = aucs_diff$auc_diff.x - aucs_diff$auc_diff.y
aucs_diff %>% ggplot(aes(x=diff_of_diffs)) + geom_density() + geom_jitter(aes(y=0), width=0, height=0.001) + ggtitle("Raw data + estimated density")
Gaussian mixture models of diffeences of AUC differences
1 Cluster
fit1 = Mclust(aucs_diff$diff_of_diffs, G=1)
print(summary(fit1, parameters=2))## ----------------------------------------------------
## Gaussian finite mixture model fitted by EM algorithm
## ----------------------------------------------------
##
## Mclust X (univariate normal) model with 1 component:
##
## log-likelihood n df BIC ICL
## -309.7775 64 2 -627.8728 -627.8728
##
## Clustering table:
## 1
## 64
##
## Mixing probabilities:
## 1
## 1
##
## Means:
## [1] 18.71731
##
## Variances:
## [1] 936.98752 Clusters
fit2 = Mclust(aucs_diff$diff_of_diffs, G=2)
print(summary(fit2, parameters=T))## ----------------------------------------------------
## Gaussian finite mixture model fitted by EM algorithm
## ----------------------------------------------------
##
## Mclust E (univariate, equal variance) model with 2 components:
##
## log-likelihood n df BIC ICL
## -304.2372 64 4 -625.11 -631.8846
##
## Clustering table:
## 1 2
## 56 8
##
## Mixing probabilities:
## 1 2
## 0.8518197 0.1481803
##
## Means:
## 1 2
## 9.268756 73.032642
##
## Variances:
## 1 2
## 423.7863 423.78633 Clusters
fit3 = Mclust(aucs_diff$diff_of_diffs, G=3)
print(summary(fit3, parameters=T))## ----------------------------------------------------
## Gaussian finite mixture model fitted by EM algorithm
## ----------------------------------------------------
##
## Mclust E (univariate, equal variance) model with 3 components:
##
## log-likelihood n df BIC ICL
## -304.2422 64 6 -633.4377 -695.7058
##
## Clustering table:
## 1 2 3
## 10 46 8
##
## Mixing probabilities:
## 1 2 3
## 0.3512637 0.5047542 0.1439820
##
## Means:
## 1 2 3
## 4.325029 12.992664 73.897965
##
## Variances:
## 1 2 3
## 409.2744 409.2744 409.2744According to the Bayesian information criterion, a model with two clusters describes the data best.
Fitted model:
aucs_diff %>%
ggplot(aes(x=diff_of_diffs)) +
geom_jitter(aes(y=0, color=first_speaker_type.x), width=0, height=0.001) +
ggtitle("Raw data + Components of gaussian mixture") +
stat_function(fun = dnorm, args = list(mean = fit2$parameters$mean[1], sd = sqrt(fit2$parameters$variance$sigmasq[1]))) +
stat_function(fun = dnorm, args = list(mean = fit2$parameters$mean[2], sd = sqrt(fit2$parameters$variance$sigmasq[2])))## Warning: Removed 101 row(s) containing missing values (geom_path).
Compute likelihoods based on the adaptation model
## Generalized linear mixed model fit by maximum likelihood (Laplace
## Approximation) [glmerMod]
## Family: binomial ( logit )
## Formula: most_likely_model ~ condition + test_order + first_speaker_type +
## first_speaker_type * condition + (1 | workerid)
## Data: d.post_test
##
## AIC BIC logLik deviance df.resid
## 154 171 -71 142 118
##
## Scaled residuals:
## Min 1Q Median 3Q Max
## -1.6870 -0.5887 -0.1792 0.5928 1.7182
##
## Random effects:
## Groups Name Variance Std.Dev.
## workerid (Intercept) 2.34 1.53
## Number of obs: 124, groups: workerid, 62
##
## Fixed effects:
## Estimate Std. Error z value
## (Intercept) -0.4767 0.3321 -1.435
## conditioncautious -1.2705 0.3757 -3.382
## test_orderparallel -0.4084 0.3236 -1.262
## first_speaker_typecautious 0.4631 0.3306 1.401
## conditioncautious:first_speaker_typecautious 0.4238 0.2719 1.559
## Pr(>|z|)
## (Intercept) 0.15123
## conditioncautious 0.00072 ***
## test_orderparallel 0.20701
## first_speaker_typecautious 0.16130
## conditioncautious:first_speaker_typecautious 0.11899
## ---
## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
##
## Correlation of Fixed Effects:
## (Intr) cndtnc tst_rd frst__
## conditincts 0.298
## tst_rdrprll 0.134 0.245
## frst_spkr_t -0.221 -0.289 -0.100
## cndtncts:__ -0.217 -0.402 -0.122 0.213## Linear mixed model fit by REML. t-tests use Satterthwaite's method [
## lmerModLmerTest]
## Formula: likelihood_ratio ~ condition + test_order + first_speaker_type +
## first_speaker_type * condition + (1 | workerid)
## Data: d.post_test
##
## REML criterion at convergence: 1646
##
## Scaled residuals:
## Min 1Q Median 3Q Max
## -2.0346 -0.6043 -0.1376 0.4363 2.5545
##
## Random effects:
## Groups Name Variance Std.Dev.
## workerid (Intercept) 14000 118.3
## Residual 36711 191.6
## Number of obs: 124, groups: workerid, 62
##
## Fixed effects:
## Estimate Std. Error df
## (Intercept) 4.049 22.904 59.000
## conditioncautious -74.331 17.242 60.000
## test_orderparallel -6.180 22.856 59.000
## first_speaker_typecautious 41.298 22.892 59.000
## conditioncautious:first_speaker_typecautious 21.163 17.242 60.000
## t value Pr(>|t|)
## (Intercept) 0.177 0.8603
## conditioncautious -4.311 6.14e-05 ***
## test_orderparallel -0.270 0.7878
## first_speaker_typecautious 1.804 0.0763 .
## conditioncautious:first_speaker_typecautious 1.227 0.2245
## ---
## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
##
## Correlation of Fixed Effects:
## (Intr) cndtnc tst_rd frst__
## conditincts 0.000
## tst_rdrprll 0.032 0.000
## frst_spkr_t -0.065 0.000 -0.002
## cndtncts:__ 0.000 -0.065 0.000 0.000## Linear mixed model fit by REML. t-tests use Satterthwaite's method [
## lmerModLmerTest]
## Formula: likelihood_ratio ~ condition + test_order + first_speaker_type +
## prior_likelihood_ratio + first_speaker_type * condition +
## (1 | workerid)
## Data: d.post_test
##
## REML criterion at convergence: 1647.4
##
## Scaled residuals:
## Min 1Q Median 3Q Max
## -1.9845 -0.5693 -0.1259 0.4594 2.6164
##
## Random effects:
## Groups Name Variance Std.Dev.
## workerid (Intercept) 14379 119.9
## Residual 36711 191.6
## Number of obs: 124, groups: workerid, 62
##
## Fixed effects:
## Estimate Std. Error df
## (Intercept) 12.72944 27.71509 58.00000
## conditioncautious -74.33107 17.24211 60.00000
## test_orderparallel -1.73800 24.30395 58.00000
## first_speaker_typecautious 41.68218 23.03583 58.00000
## prior_likelihood_ratio 0.09466 0.16801 58.00000
## conditioncautious:first_speaker_typecautious 21.16343 17.24211 60.00000
## t value Pr(>|t|)
## (Intercept) 0.459 0.6477
## conditioncautious -4.311 6.14e-05 ***
## test_orderparallel -0.072 0.9432
## first_speaker_typecautious 1.809 0.0756 .
## prior_likelihood_ratio 0.563 0.5753
## conditioncautious:first_speaker_typecautious 1.227 0.2245
## ---
## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
##
## Correlation of Fixed Effects:
## (Intr) cndtnc tst_rd frst__ prr_l_
## conditincts 0.000
## tst_rdrprll 0.206 0.000
## frst_spkr_t -0.037 0.000 0.008
## prr_lklhd_r 0.556 0.000 0.324 0.030
## cndtncts:__ 0.000 -0.065 0.000 0.000 0.000## Data: d.post_test
## Models:
## model1: likelihood_ratio ~ condition + test_order + first_speaker_type + first_speaker_type * condition + (1 | workerid)
## model2: likelihood_ratio ~ condition + test_order + first_speaker_type + prior_likelihood_ratio + first_speaker_type * condition + (1 | workerid)
## npar AIC BIC logLik deviance Chisq Df Pr(>Chisq)
## model1 7 1699.3 1719.0 -842.64 1685.3
## model2 8 1700.9 1723.5 -842.47 1684.9 0.3384 1 0.5608
List of adapters:
| workerid | first_speaker_type | test_order | noticed_manipulation | cautious_count | confident_count | aligned_count | first_adaptation_speaker_count |
|---|---|---|---|---|---|---|---|
| 1507 | cautious | reverse | 0 | 1 | 1 | 2 | 1 |
| 1513 | confident | parallel | 1 | 1 | 1 | 2 | 1 |
| 1522 | confident | reverse | 1 | 1 | 1 | 2 | 1 |
| 1524 | confident | reverse | 0 | 1 | 1 | 2 | 1 |
| 1525 | cautious | reverse | 1 | 1 | 1 | 2 | 1 |
| 1527 | cautious | parallel | 1 | 1 | 1 | 2 | 1 |
| 1528 | confident | parallel | 1 | 1 | 1 | 2 | 1 |
| 1530 | cautious | reverse | 1 | 1 | 1 | 2 | 1 |
| 1532 | cautious | parallel | 0 | 1 | 1 | 2 | 1 |
| 1537 | confident | parallel | 1 | 1 | 1 | 2 | 1 |
| 1539 | cautious | reverse | 1 | 1 | 1 | 2 | 1 |
| 1541 | cautious | parallel | 1 | 1 | 1 | 2 | 1 |
| 1543 | confident | reverse | 0 | 1 | 1 | 2 | 1 |
| 1545 | confident | parallel | 0 | 1 | 1 | 2 | 1 |
| 1547 | cautious | reverse | 1 | 1 | 1 | 2 | 1 |
| 1548 | cautious | parallel | 1 | 1 | 1 | 2 | 1 |
| 1551 | confident | reverse | 0 | 1 | 1 | 2 | 1 |
| 1552 | confident | parallel | 1 | 1 | 1 | 2 | 1 |
| 1553 | confident | parallel | 0 | 1 | 1 | 2 | 1 |
| 1558 | confident | reverse | 1 | 1 | 1 | 2 | 1 |
| 1559 | confident | reverse | 1 | 1 | 1 | 2 | 1 |
| 1560 | confident | parallel | 1 | 1 | 1 | 2 | 1 |
| 1564 | cautious | parallel | 0 | 1 | 1 | 2 | 1 |
| 1566 | confident | reverse | 1 | 1 | 1 | 2 | 1 |
| 1571 | cautious | reverse | 0 | 1 | 1 | 2 | 1 |
List of reverse adapters:
| workerid | first_speaker_type | test_order | noticed_manipulation | cautious_count | confident_count | aligned_count | first_adaptation_speaker_count |
|---|---|---|---|---|---|---|---|
| 1521 | cautious | parallel | 0 | 1 | 1 | 0 | 1 |
| 1546 | cautious | reverse | 0 | 1 | 1 | 0 | 1 |